Mixture of Inference Networks for VAE-Based Audio-Visual Speech Enhancement

We address unsupervised audio-visual speech enhancement based on variational autoencoders (VAEs), where the prior distribution of clean spectrogram is simulated using an encoder-decoder architecture. At (test) time, trained generative model (decoder) combined with a noise whose parameters need to be estimated. The initialization latent variables describing process via decoder, crucial, as overall inference problem non-convex. This usually done by output encoder given noisy audio and visual data input. Current VAE models do not provide effective because two modalities are tightly coupled (concatenated) in associated architectures. To overcome this issue, we introduce mixture networks autoencoder (MIN-VAE). Two input, respectively, data, posterior modeled Gaussian distributions from each encoder. variable also latent, therefore learning optimal balance between encoders well. By training shared network learns adaptively fuse modalities. Moreover, at test encoder, taking (clean) used for initialization. A approach derived train proposed model. Thanks novel procedure robust initialization, MIN-VAE exhibits superior performance than standard audio-only well counterparts.